Open a browser window for search.acs.org in DEVONagent Pro or any other web browser. Search for "mercury".

At the bottom of the results, Control-click the Next link and copy the link. Leave this window open while developing the plugin.

Choose File > New Plugin. This will open the Plugins & Scanners window. In the URL field, paste the copied link, and replace mercury in the URL with _agentQuery_. The _agentQuery_ placeholder is replaced with your search terms when using the plugin. An _agentOffset_ placeholder, used by DEVONagent Pro to jump from result page to result page, is automatically inserted since a start= parameter is present. This would also be used with other such parameters, like next=. Enter a name for the plugin and optionally, a description. The Search Engine radio button is automatically selected by DEVONagent Pro. Then press the Add button.

Let's test the plugin. Press the Test button at the top of the Plugins & Scanners window. The results show the created plugin is functioning but is causing unnecessary traffic and returning many unnecessary results. Select any unwanted links in the Log, e.g., Twitter, Facebook, mailto, etc. Control-click one, and choose Exclude Domain.

Also, select one of the links with search.acs.org in its URL and exclude that domain as well. This keeps search pages from becoming possible results. Note you can sort the results by location to help isolate results by location for easier handling. Then press the Retry button. You should see 10 links being reported for the search URLs in the log.

Now choose File > New Search. Select our new plugin from the options under the magnifying glass. Run a test search using the search term mercury. We are getting some results but more fine-tuning is needed for better returns.

When we examine the URL of the results in the browser window, we can see they are from www.acs.org/content. We can use this information to tell DEVONagent Pro to only match items from that domain. In the Plugins & Scanners window, double-click the new plugin. The code for it will be displayed and ready for editing. Type these elements exactly as shown:

Also make sure the OffsetPerPage key is set to 10. This ensures DEVONagent Pro will go the subsequent pages of results.

Running a new search, we get some results. We're not quite there but we are targeting the correct domain!

Now let's use another powerful set of keys: TextStart and TextEnd. These tell DEVONagent Pro to try and make its text matches within the boundaries you've set in the plugin.

If you have enabled Developer Extras in Preferences > Web, you can Control- or right-click parts of the rendered page, e.g., the title of the article, and choose Inspect Element. Also, select items in the markup in the developer tools and the item on the rendered page will be highlighted. Otherwise, you can select Web > Source to show the HTML source and search for items in the content of the page.

Inspect or find the title of the article. In this case, notice it's an <h1> element. Then scroll down and inspect or find the paragraphs after the main article. We see this section <class="reference parbase"> but this may appear more than once on the page. Looking a little further down, we see a section using <div class="ui-nav">. As this is the only place this element appears and we feel confident the footer appears on each result page, we will use this value. Type these XML elements exactly as shown:

Now let's give this another try. We rerun the mercury search again and this time it yields many more good results!

We now have a working plugin for the acs.org site. You can check both the results and digest panes in the Search window to further fine tune your plugin. Below are some useful keys used to more accurately target elements or properties, e.g., the date, from the resulting pages.

As mentioned above, you can examine the HTML source code of results and try to find strings that can be used for these keys. When looking at the topics in the digest pane after you've done your searches, you can set the NoTopics list. Any word that should not be included as a topic can be added to this list. Select words to exclude in the Digest tab and press ⌘C to copy them. You can then paste them into a <string>, as one comma-delimited line, like so…

In other cases, you might also want to look at the source of the plugin and validate the values for Start, OffsetPerPage, and ResultsPerPage, etc. You can also add a CrawlDelay key to slow the pace of contact with the search engine's servers. This can help avoid overloading it and make DEVONagent Pro appear to be a bit more like a human is searching. Read more...

In addition, DEVONagent Pro may not always be able to insert the _agentOffset_ and _agentNumber_ placeholders automatically. You might have to do this on your own; otherwise, only the first results page is used.

Finally, uncheck any operators the search engine doesn't support.

With the steps described in this tutorial, you should be able to go through the steps of writing a plugin yourself. With a bit of practice, it is possible to write an optimized plugin that will deliver good results for DEVONagent Pro to work with. It might help to look at the additional plugins that already come with DEVONagent Pro before designing your own plugin. For the built-in plugins, Control- or right-click and duplicate a plugin. You can then open and inspect its internal code.

And remember: If you think your plugin is great for others, please share it with us!